feat: add kubelet systemd service hardening option #9194

alegrey91 · 2022-08-19T09:38:50Z

What type of PR is this?

/kind feature

What this PR does / why we need it:

Improve security of kubelet service, minimizing access through systemd.

Which issue(s) this PR fixes:

Fixes #9107

Special notes for your reviewer:

Does this PR introduce a user-facing change?:

Add kubelet systemd service hardening option `kubelet_systemd_hardening: [true|false]`

k8s-ci-robot · 2022-08-19T09:38:58Z

Hi @alegrey91. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

cristicalin · 2022-08-22T07:49:51Z

roles/kubernetes/node/templates/kubelet.service.j2

+{% if kubelet_systemd_hardening %}
+# Hardening setup
+IPAddressDeny=any
+IPAddressAllow={{ groups['kube_control_plane'] | map('extract', hostvars, ['ansible_host']) | join(' ') }}


We may be dealing with multi-homed hosts which have dedicated management interfaces (think ssh access only) and use a separate interface for kubelet traffic. This hardening setup should allow to bind to dedicated interfaces IMHO.

Hi @cristicalin,
tell me know if I understand correctly.
Actually you don't need to bind a dedicated interface for kubelet.
The systemd service set a dedicated firewall just for the running binary (in this case kubelet).
If you try to access ssh, you are free to do this because the firewall does not intervene on the ssh incoming packets.

Here's the example:
From the control plane node, we run a python -m http.server 9999 (not managed by the systemd service), and from an external machine, we run this:

curl --max-time 5 http://10.0.126.110:9999/ <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd"> <html> <head> <meta http-equiv="Content-Type" content="text/html; charset=utf-8"> <title>Directory listing for /</title> </head> <body> <h1>Directory listing for /</h1> <hr> <ul> </ul> <hr> </body> </html>

Then, always from the control plane node, we start the systemd service for kubelet. From an external machine we run this:

curl --max-time 5 -k https://10.0.126.110:10250/pods/ curl: (28) Connection timed out after 5001 milliseconds

As you can see, in the second example, the systemd firewall is intervening only on kubelet service, blocking the incoming packets. The other packets, instead, are not interested by the firewall.

What I meant was on this line:

{{ groups['kube_control_plane'] | map('extract', hostvars, ['ansible_host']) | join(' ') }}

You are making the assumption that only the main interface is used where the deployer may want to have the kubelet interface set to a different interface. ansible_host is the IP on which the ansible controller connects to the host which may not be the IP that the deployer wants to use for this type of traffic.

Ok, now I got your point.
So, essentially we should list the IPs used by the kube-apiserver, instead of the ansible_host variable.
This because the kube-apiserver is the only component that communicates with the kubelet.

@cristicalin do you have suggestions on how to retrieve this information? I'm stuck here

What if we set the kubelet_secure_address as {{ groups['kube_control_plane'] | map('extract', hostvars, ['ansible_host'] | join(' ') }} by default, so the user should not deal with this variable in a common case. Instead, when needed, it can be changed.
In case the user bound the kube-apiserver to a different interface (that is not the one used by ansible), it should just define the variable in this way: kubelet_secure_address="172.16.23.10 172.16.23.11 172.16.23.12".

You should be mindful of variable precedence if doing that but I'd like to see an example of how the code would look like.

I updated the code. Take a look and let me know what do you think :)

The variable that you suggested is defined like so:

kubelet_secure_address: "{{ groups['kube_control_plane'] | map('extract', hostvars, ['ansible_host']) | join(' ') }}"

You can just override it in this way:

kubelet_secure_address: "172.16.23.10 172.16.23.11 172.16.23.12"

Sorry @cristicalin, did you take a look at the code?

oomichi · 2022-08-23T09:32:33Z

Nice work

/ok-to-test
/approve

k8s-ci-robot · 2022-08-23T09:32:46Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: alegrey91, oomichi

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [oomichi]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

cristicalin · 2022-08-28T13:00:14Z

docs/hardening.md

+# in case you have multiple interfaces in your
+# control plane nodes and you want to specify the right
+# IP addresses
+kubelet_secure_address: "192.168.10.110 192.168.10.111 192.168.10.112"


The variable name is singular and the example is in the plural, this is confusing, the example should make it clear how to set these either on a per node basis and change the way the list is compiled or rename the variable to the plural and explain that the variable is a list of addresses from the secure interface of each node.

Yes, probably the variable should be renamed in kubelet_secure_addresses.

docs/vars.md

cristicalin · 2022-08-28T13:08:51Z

Since this patch can affect the way kube-apiserver talks to the kubelets, I think it would be nice to include a diagram explaining the connectivity flow and how it can be moved to a dedicated interface.

alegrey91 · 2022-08-28T16:18:57Z

Since this patch can affect the way kube-apiserver talks to the kubelets, I think it would be nice to include a diagram explaining the connectivity flow and how it can be moved to a dedicated interface.

Yes, I agree. Where should I insert the diagram?
What to you think about this?

cristicalin · 2022-08-30T05:48:50Z

Hi @alegrey91 , please rebase this PR on latest state of the master branch to be able to pass the CI tests.

You can place the diagram as an image in docs/img and inline it in hardening.md.

Co-authored-by: Cristian Calin <6627509+cristicalin@users.noreply.github.com>

alegrey91 · 2022-08-30T06:40:35Z

Hi @cristicalin, I just rebased and added the diagram. Thanks for your help with the PR :)

cristicalin · 2022-08-30T18:15:13Z

Thanks @alegrey91

/lgtm

alegrey91 · 2022-08-30T19:48:04Z

Thanks @cristicalin and @oomichi

[CI] remove opensuse Leap from molecule test blocking CI (kubernetes-sigs#9229) Fixes for calico_datastore: etcd (kubernetes-sigs#9228) It seems that PR kubernetes-sigs#8839 broke `calico_datastore: etcd` when it removed ipamconfig support for etcd mode. This PR fixes some failing tasks when `calico_datastore == etcd`, but it does not restore ipamconfig support for calico in etcd mode. If someone wants to restore ipamconfig support for `calico_datastore: etcd` please submit a follow up PR for that. kube-vip shoud fail if kube_proxy_strict_arp is false in arp mod (kubernetes-sigs#9223) * fix-kube-vip-strict-arp * fix-kube-vip-strict-arp add runc v1.1.4 (kubernetes-sigs#9230) Fix typo. Fix kube_ovn_hw_offload value (kubernetes-sigs#9218) Fix cloud_init files for different distros (kubernetes-sigs#9232) Fix abort because calicoctl.sh is not a full path (kubernetes-sigs#9217) feat: add kubelet systemd service hardening option (kubernetes-sigs#9194) * feat: add kubelet systemd service hardening option * refactor: move variable name to kubelet_secure_addresses Co-authored-by: Cristian Calin <6627509+cristicalin@users.noreply.github.com> * docs: add diagram about kubelet_secure_addresses variable Co-authored-by: Cristian Calin <6627509+cristicalin@users.noreply.github.com> make calico installation more stable (kubernetes-sigs#9227) Signed-off-by: hang.jiang <hang.jiang@daocloud.io> Signed-off-by: hang.jiang <hang.jiang@daocloud.io> Update security contacts file (kubernetes-sigs#9235) disable kubelet_authorization_mode_webhook by default (kubernetes-sigs#9238) add-yankay-to-reviewers (kubernetes-sigs#9247)

) * feat: add kubelet systemd service hardening option * refactor: move variable name to kubelet_secure_addresses Co-authored-by: Cristian Calin <6627509+cristicalin@users.noreply.github.com> * docs: add diagram about kubelet_secure_addresses variable Co-authored-by: Cristian Calin <6627509+cristicalin@users.noreply.github.com>

k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Aug 19, 2022

k8s-ci-robot added needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels Aug 19, 2022

k8s-ci-robot requested review from bozzo and oomichi August 19, 2022 09:39

cristicalin reviewed Aug 22, 2022

View reviewed changes

alegrey91 requested review from cristicalin and removed request for oomichi and bozzo August 22, 2022 09:47

alegrey91 force-pushed the master branch from e7a04fc to 3de3728 Compare August 23, 2022 08:26

k8s-ci-robot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. and removed size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels Aug 23, 2022

alegrey91 force-pushed the master branch 2 times, most recently from 4f643e7 to c15d5b7 Compare August 23, 2022 09:12

k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Aug 23, 2022

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Aug 23, 2022

cristicalin reviewed Aug 28, 2022

View reviewed changes

docs/vars.md Outdated Show resolved Hide resolved

cristicalin reviewed Aug 28, 2022

View reviewed changes

docs/vars.md Outdated Show resolved Hide resolved

alegrey91 force-pushed the master branch from 1d7dfc4 to 1877c5e Compare August 28, 2022 16:25

alegrey91 requested a review from cristicalin August 28, 2022 16:25

alegrey91 force-pushed the master branch from 1877c5e to c1a2c5b Compare August 29, 2022 16:07

k8s-ci-robot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Aug 29, 2022

alegrey91 force-pushed the master branch from c1a2c5b to c15d5b7 Compare August 29, 2022 16:25

k8s-ci-robot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. and removed size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Aug 29, 2022

alegrey91 and others added 2 commits August 30, 2022 08:33

feat: add kubelet systemd service hardening option

026f477

refactor: move variable name to kubelet_secure_addresses

83cc772

Co-authored-by: Cristian Calin <6627509+cristicalin@users.noreply.github.com>

alegrey91 force-pushed the master branch from 3cdeb7f to 83cc772 Compare August 30, 2022 06:33

docs: add diagram about kubelet_secure_addresses variable

10c08a4

k8s-ci-robot assigned cristicalin Aug 30, 2022

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Aug 30, 2022

k8s-ci-robot merged commit acb6f24 into kubernetes-sigs:master Aug 30, 2022

floryut mentioned this pull request Sep 19, 2022

Release Proposal v2.20 #9291

Closed

dgerhardt mentioned this pull request Sep 28, 2022

Upgrade fails when ansible_host is not set for control plane nodes #9344

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: add kubelet systemd service hardening option #9194

feat: add kubelet systemd service hardening option #9194

alegrey91 commented Aug 19, 2022 •

edited by floryut

Loading

k8s-ci-robot commented Aug 19, 2022

cristicalin Aug 22, 2022

alegrey91 Aug 22, 2022 •

edited

Loading

cristicalin Aug 22, 2022

alegrey91 Aug 22, 2022 •

edited

Loading

alegrey91 Aug 22, 2022

alegrey91 Aug 23, 2022 •

edited

Loading

cristicalin Aug 24, 2022

alegrey91 Aug 24, 2022

alegrey91 Aug 24, 2022

alegrey91 Aug 25, 2022

oomichi commented Aug 23, 2022

k8s-ci-robot commented Aug 23, 2022

cristicalin Aug 28, 2022 •

edited

Loading

alegrey91 Aug 28, 2022

cristicalin commented Aug 28, 2022

alegrey91 commented Aug 28, 2022

cristicalin commented Aug 30, 2022

alegrey91 commented Aug 30, 2022

cristicalin commented Aug 30, 2022

alegrey91 commented Aug 30, 2022

feat: add kubelet systemd service hardening option #9194

feat: add kubelet systemd service hardening option #9194

Conversation

alegrey91 commented Aug 19, 2022 • edited by floryut Loading

k8s-ci-robot commented Aug 19, 2022

Choose a reason for hiding this comment

alegrey91 Aug 22, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alegrey91 Aug 22, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alegrey91 Aug 23, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

oomichi commented Aug 23, 2022

k8s-ci-robot commented Aug 23, 2022

cristicalin Aug 28, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cristicalin commented Aug 28, 2022

alegrey91 commented Aug 28, 2022

cristicalin commented Aug 30, 2022

alegrey91 commented Aug 30, 2022

cristicalin commented Aug 30, 2022

alegrey91 commented Aug 30, 2022

alegrey91 commented Aug 19, 2022 •

edited by floryut

Loading

alegrey91 Aug 22, 2022 •

edited

Loading

alegrey91 Aug 22, 2022 •

edited

Loading

alegrey91 Aug 23, 2022 •

edited

Loading

cristicalin Aug 28, 2022 •

edited

Loading